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EXECUTIVE  SUMMARY 


The  purpose  of  this  study  was  to  determine  the  impact  of  increased 
emphasis  on  test  and  evaluation  on  the  acquisition  of  the  U.S.  Navy 
S-3A  antisubmarine  warfare  aircraft.  The  acquisition  project  had  been 
in  being  almost  four  years  and  in  contract  al most  two  years  when  the 
Secretary  of  Defense  promulgated  explicit  policy  relating  to  the 
initiation  of  operational  test  and  evaluation  in  the  acquisition 
process  as  early  as  possible,  on  13  July  1971.  This  requirement  was 
thus  imposed  on  the  S-3A  project.  Data  and  information  pertinent  to 
measuring  the  effect  of  early  initiation  of  operational  test  and 
evaluation  on  the  contract,  the  cost,  schedule  and  quality  and  con- 
sistency of  data  was  acquired  by  conducting  a literature  search  in  the 
Defense  Systems  Management  School  Library,  conducting  interviews 
of  various  Department  of  Defense  officials  and  an  examination  of 
unclassified  documents  held  in  the  S-3A  Project  Office. 

Some  comments  concerning  the  development  of  the  requirement  for 
incorporating  operational  test  and  evaluation  early  is  presented  as  is 
a description  of  the  problem  thus  specifically  imposed  on  the  S-3A 
project. 

After  discussing  the  effect  of  conducting  concurrent  development 
testing  and  operational  testing,  it  was  concluded  that  initiation  of 
early  operational  testing  had  no  impact  on  the  basic  contract, 
resulted  in  an  increase  in  cost  amounting  to  less  than  one  percent 


of  the  total  estimated  cost  of  acquisition  and  ownership  and  was 
accomplished  by  making  a relatively  minor  revision  to  the  original 
contractual  schedule.  It  was  also  concluded  that  a possible  lack  of 
coordination  between  test  agencies  has  resulted  in  conflicting  con- 
clusions derived  from  the  results  of  the  concurrent  testing. 
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AND  EVALUATION  ON  THE  ACQUISITION  OF  THE 
U.S.  NAVY  S-3A  ASW  AIRCRAFT 

INTRODUCTION 

The  purpose  of  this  study  was  to  determine  the  impact  of  increased 
emphasis  on  test  and  evaluation  on  the  acquisition  of  the  U.S.  Navy 
S-3A  antisubmarine  warfare  aircraft.  By  July  1971,  this  program  had  been 
in  planning  for  four  years , in  contract  for  two  years  and  was  eighteen 
months  from  first  flight.  Also  in  July  1971,  a significant  change  in 
Department  of  Defense  (DOD)  policy  concerning  the  acquisition  of 
major  defense  systems  became  effective.  This  new  policy  provided,  in 
part,  for  initiation  of  system  operational  test  and  evaluation  as  soon 
as  possible.  The  policy  statement  did  not  specifically  address  appli- 
cation to  programs  in  progress.  However,  coincident  with  promulgation 
of  this  new  policy,  the  decision  was  made  to  conduct  concurrent  develop- 
ment and  operational  test  and  evaluation  of  the  S-3A  aircraft  and  it's 
antisubmarine  warfare  avionics  suite  as  soon  as  possible.  This  decision 
required  a major  revision  of  the  Project  Master  Plan  and  impacted 
on  virtually  every  facet  of  the  program. 

The  data  and  information  germane  to  this  study  were  acquired  by 

* • \ * * 

conducting  a search  of  the  pertinent  literature  contained  in  the 
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This  study  represents  the  views,  conclusions  and  recommendations  of 
the  author  and  does  not  necessarily  reflect  the  official  opinion  of 
• the  Defense  Systems  Management  School  nor  the  Department  of  Defense. 
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Defense  Systems  Management  School  library,  by  conducting  formal  and 
informal  interviews  of  various  officials  within  the  Department  of 
Defense  and  by  examination  of  relevent  unclassified  documents  made 
available  by  the  S-3A  Project  Office.  Although  this  report  is  not 
intended  to  reflect  any  parochialism,  the  terminology  used  herein  will 
generally  be  that  common  usage  in  the  U.S.  Navy.  Also,  the  Army  and 
Air  Force  have  projects  in  progress  which  have  been  impacted  by  the 
new  test  and  evaluation  policy.  Other  than  to  confirm  the  observations 
which  resulted  from  the  study  of  the  Navy's  S-3A  project, and  then 
only  through  informal  contact  with  a member  of  another  service  who 
was  engaged  in  such  a project,  no  attempt  was  made  to  study  in  depth 
any  project  except  the  S-3A. 

The  impact  of  conducting  early  operational  test  and  evaluation 
on  the  S-3A  weapon  was  determined,  primarily  subjectively,  by  analyzing 
the  effect  on  the  system  acquisition  contract, on  cost  and  on  schedule 
and  by  evaluating  the  quality  and  consistency  of  the  results  of  the 
concurrent  test  and  evaluation  program. 
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THE  REQUIREMENT 

The  names  of  most  prominent  critics  of  the  Department  of 
Defense  (DOD)  are  familiar  to  most  of  those  associated  with,  and 
those  within,  DOD  since  the  mid-1960's.  Much  of  this  public  criticism 
focused  on  DOD's  management  and  methods  of  acquiring  major  weapon 
systems.  In  19&9,  'the  Secretary  of  Defense,  who  was  also  critical  but 
not  publicly  vituperous , commissioned  a Blue  Ribbon  Defense  Panel  to 
assess  the  DOD’s  acquisition  management  policy  and  procedures  and  to 
make  recommendations  for  improvements.  A section  of  the  Panel's 
1970  Report^ pertained  to  test  and  evaluation.  Some  relevent 
exerpts  from  this  Report  are: 

"It  has  been  customary  to  think  of  operational  test  and  evaluation 
in  terms  of  physical  testing  (under  various  designations  such  as  operational 
suitability  testing,  employment  testing  or  field  experimentation). 
Operational  testing  is  a very  important  activity  (which  has  often  been 
done  poorly),  but  it  is  emphasized  that  the  goal  is  operational  evalu- 
ation and  that  physical  testing  is  only  one  means  of  performing  operational 
evaluation.  This  is  an  important  point  since  it  is  often  argued  that 
operational  testing  must  await  production  of  an  adequate  number  of 
operationally-configured  systems;  and  by  this  time  it  is  too  late  to 
use  the  information  gathered  to  help  decide  whether  to  produce  the  new 
system  or  even  to  influence  in  any  significant  way  the  nature  of  the 
system  procured." 

"...an  effective  program  of  operational  test  and  evaluation  must 
start  with  the  earliest  requirements  and  continue. . .for  the  life  of  the 
system.  It  should  be  pointed  out  that  to  conduct  this  lifetime  of 
operational  test  and  evaluation  it  is  not  necessary  to  conduct  a different 
type  of  testing  to  fulfill  each  purpose  and  satisfy  each  'customer'.  There 
is  a great  deal  of  overlapping,  and  in  large  measure  a generalized 
program  of  operational  test  and  evaluation  can  assist  with  all  purposes." 

Responding  to  the  Panel's  recommendations,  the  Secretary  of  Defense 
(SECDEF)  promulgated  broad  policy  concerning  the  acquisition  of  major 


* Superscript  numbers  in  parentheses  refer  to  references  listed  at 
the  end  of  this  report.  See  the  Table  of  Contents  for  page  number. 
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defense  systems  in  July  1971 • His  guidance  concerning  test  and 
evaluation  was  explicit. 

"Test  and  evaluation  shall  commence  as  early  as  possible.  A 
determination  of  the  operational  suitability,  including  logistic 
support  requirements,  will  be  made  prior  to  large  scale  production, 
commitments,  making  use  of  the  most  realistic  test  environment 
possible  and  the  best  representation  of  future  operational  systems 
available.  The  results  of  this  operational  testing  will  be  evaluated 
and  presented  to  the  Defense  Systems  Acquisition  Review  Council  at 
the  time  of  the  production  decision." 

Historically,  in  the  services  and  the  Navy  in  particular, 
operational  test  and  evaluation  has  been  downstream  of  any  major  decision 
point  in  the  acquisition  process.  As  such,  operational  test  and 
evaluation  has  been  directed  more  to  determining  how  suitable  the 
system  or  equipment  was  and,  given  it's  determined  degree  of  suita- 
bility, how  best  to  use  it  rather  than  contributing  to  the  decision 
whether  the  system  or  equipment  should  be  acquired  in  the  first 
place.  Thus,  operational  test  and  evaluation  generally  followed  the 
decision  to  go  into  production  and  was  conducted  just  prior  to  or 
simultaneously  with  fleet  introduction.  In  all  too  many  instances, 
operational  test  and  evaluation  was  then  accomplished  in  an  environ- 
ment or  under  conditions  which  had  little  resemblance  to  actual 
fleet  use  and  was  often  conducted  by  an  agency  which  was  at  least 
restricted  in  its  concept  of  fleet  utilization.  The  Blue  Ribbon 
Defense  Panel  recognized  these  shortcomings.  It  saw  that  the 
early  and  proper  application  of  operational  test  and  evaluation  could 
be  a powerful  tool  to  be  used  in  deciding  whether  the  development  cf 
a system  warranted  proceeding  to  a subsequent  phase  of  the  acquisition 
process.  SECDEF  was  explicit  in  his  concurrence  in  the  Panels' 


recommendation. 
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Policy,  to  be  effective,  must  be  adequately  implemented. 

Shortly  before  SECDEF  released  his  acquisition  policy  directive, 
he  established  the  Deputy  Director  of  Defense  Research  and  Engineering 
(Test  and  Evaluation)  (DDT&E).^  The  latter  has  been  tasked  to 
monitor  all  test  and  evaluations  conducted  by  the  services  for  major 
defense  weapon  systems  and  any  other  programs  as  considered  necessary. 

He  reports  the  results  of  his  assessment  of  the  adequacy  and  suitability 
of  operational  test  and  evaluation  of  a system  under  development  di- 
rectly to  SECDEF  as  well  as  to  the  Defense  Systems  Acquisition  Review 
Council  whenever  an  acquisition  decision  point  has  been  reached.  With 
such  assigned  responsibilities,  an  aggressive  DDT&E  exerts  considerable 
influence  on  SECDEF' s decision  to  proceed  from  one  acquisition  phase 
to  the  next. 

Additional  impetus  in  regards  to  the  requirement  for  test  and 
evaluation  of  the  S-3A  project  resulted  from  receipt  of  a Government 
Accounting  Office  (GAO)  Report  dated  29  March  1971  , which  was 

made  available  to  the  Office  of  the  Chief  of  Naval  Operations  (CNO) 
on  16  July  1971^-  In  a section  on  matters  for  consideration  by  the 
Preparedness  Investigating  Subcommittee,  Senate  Committee  on  Armed 
Services,  the  GAO  recommended  that  "consideration  should  be  given  to 
conducting  an  assessment  of  the  S-3A,  independent  of  program  manage- 
ment, prior  to  the  production  decision".  Although  CNO's  formal 
response  ^ addressed  the  assignment  of  qualified  witnesses  to 
observe  significant  initial  development  test  and  evaluation,  it  is 
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known  that  he  informally  advised  the  subcommittee  that  a major 
revision  of  the  test  and  evalxiation  program  for  the  S-3A  project  was 


THE  PROBLEM 


The  contract  for  the  design,  development  and  encremen"cal  pro- 
duction of  the  S-3A  weapon  system  was  awarded  on  1 August  1969  • It 
is  a multibillion  dollar,  multiyear,' fixed  price  incentive  contract 
of  considerable  complexity,  containing  many  interdependent  and  inter- 
related clauses.  The  signing  of  the  contract  was  the  culmination  of 
much  planning,  design  and  development  effort  and  negotiation.  A 
significant  feature  of  the  contract  is  the  incorporation  of  five 
specific  milestones  which  must  be  satisfactorily  completed  prior  to 
proceeding  to  the  next  phase  of  the  acquisition  project.  A summary 
of  the  contract  schedule,  showing  these  milestones,  is  presented  in 
Figure  1.  One  of  the  first  two  milestones,  is  a laboratory  demonstration 
of  the  complete  antisubmarine  warfare  avionics  suite.  The  other  is  a 
thirty  minute  "first  flight"  of  the  aircraft  less  the  avionics  suite. 
These  two  milestones  provided  the  basis  for  the  initial  production 
decision  scheduled  for  not  later  than  1 April  1972  and  for  the  initiation 
of  Navy  development  testing.  In  conjunction  with  development  testing, 
a Navy  Preliminary  Evaluation  (NPE)  * and  a Navy  Preliminary  Appraisal 
(NPA)  were  each  designated  contractual  milestones.  The  NPE  provided 
the  basis  for  the  full  scale  production  decision  to  occur  not  later 
than  1 August  1972  and  the  NPA  signaled  the  readiness  to  commence 
Board  of  Inspection  and  Survey  (BIS)  trials  on  9 October  1973.  The 

* The  reader  is  requested  to  refer  to  the  Glossary  for  definitions 
of  Navy  test  and  evaluation  related  terminology  as  used  herein. 
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final  milestone  was  the  delivery  of  a production  configured  complete 
aircraft  and  avionic  system  for  the  commencement  of  the  BIS  trials . 
Completion  of  this  milestone  was  seen  as  confirmation  of  the  technical 
and  schedule  adequacy  of  the  project  as  it  entered  into  full  scale 
production.  Operational  test  and  evaluation  was  scheduled  to  commence 
on  15  March  197^ , nearly  a year  after  the  decision  to  go  into  full 
scale  production  had  been  made. 

After  two  years  of  progress  under  the  terms  of  the  contract, 
the  Chief  of  Naval  Operations  (CNO),  responding  to  a strong  recommen- 
dation by  DDT&E,  directed  that  operational  test  and  evaluation  of 
the  S-3A  system  be  initiated  as  "early  as  the  schedule  would  permit." ^ 
CNO  directed  further,  that  there  should  be  no  significant  cost  increase 
or  schedule  delay  associated  with  early  initiation  of  operational 
test  and  evlauation.  And  finally,  he  directed  that  the  operational 
evaluation  (operational  test  and  evaluation  ) be  conducted  by  the 
Commander,  Operational  Test  and  Evaluation  Force  ( COMOPTEVFOR ) . 

Thus,  in  August  1972,  the  S-3A  Project  Manager  was  faced  with 
a major  revision  of  his  contract  schedule  for  the  purpose  of  accommodating 
early  initiation  of  test  and  evaluation  while  constrained  by  the 
contract,  cost  and  schedule.  An  additional  set  of  constraints  were 
added;  operational  test  and  evaluation  was  to  be  accomplished  by  an 
independent  agency  and  the  results  of  the  operational  test  and 
evaluation  would  be  used  by  DDT&E  in  a recommendation  to  proceed  into 
full  scale  production.  This,  therefore,  is  the  problem  which  is 
the  basis  for  this  study  and  report  effort. 


DISCUSSION  AND  ANALYSIS 
The  Contract 

The  most  significant  aspect  of  the  problem — accommodating  for 
the  increased  emphasis  on  operational  test  and  evaluation — confronting 
the  S--3A  Project  Manager  in  August  1972,  was  the  fragilty  of  the  basic 
contract.  Although  the  contract  had  been  in  negotiations  for  almost 
a year  prior  to  award  on  1 August  1969,  during  which  it  had  had  the 
benefit  of  thorough  planning  and  was  subjected  to  more  than  considerable 
evaluation  by  both  the  contractor  and  the  government,  various  small 
difficulties  bagan  to  surface  shortly  thereafter.  In  sum,  the  con- 
tractor had  to  bear  the  weight  of  these  difficulties;  inflation,  wage 
increases,  increased  material  costs,  scarcities  in  some  basic  skills, 
problems  which  culminated  in  minor  schedule  slippage  passing  from 
subcontractors  who  were  faced  with  the  same  difficulties  as  the  prime 
contractor.  Since  none  of  these  difficulties,  individually,  had 
resulted  in  major  problems  for  the  contractor,  which  is  taken  as 
evidence  of  thorough  and  adequate  planning,  the  aggregate  was  such 
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that  the  contractor  was  tending  to  be  guardedly  defensive  by  August 
1971.  This  was  well  known  in  the  Project  Office. 

One  of  the  options  the  Project  Manager  considered  in  an  effort 
to  initiate  operational  test  and  evaluation  earlier  than,  originally 
scheduled  was  to  renegotiate  the  contract.  He  stated,  however,  that 
after  considering  all  the  ramifications,  this  option  was  discarded 
due  to  its  effect  on  the  cost  of  acquisition.  The  contractor  had 
by  then  two  years  of  experience  with  the  contract  and  could  be 
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expected  to  learn  from  this  experience.  This  learning,  it  was 
believed,  would  be  translated  into  increased  cost.  That  the  Project 
Office  was  seriously  concerned  about  the  fragilty  of  the  contract 
is  evidenced  by  reference  to  the  subject  in  numerous  letters  and 
memoranda,  an  example  of  which  from  a Memorandum  for  Record  is:^^ 

"If  such  additional  tests  [operational  test  and  evaluation]  are 
forced  on  the  program  they  will  more  than  likely  result  in  fracturing 
the  contract  and  cause  a delta  cost  far  in  excess  of  the  previously 
noted. " 

The  following  is  quoted  from  another  Memorandum  for  Record^®); 

"...it  must  be  remembered  that  in  every  case  the  new  testing  to 
be  introduced  is  to  be  only  to  the  extent  that  it  does  not  cause  a 
substantial  impact  on  the  S-3A  program,  or  give  cause  for  the  con- 
tractor to  claim  interference  to  the  extent  that  the  contract  has  been 
violated" 

As  a result  of  the  initial  assessment  of  the  impact  of  providing 
for  early  operational  test  and  evaluation,  it  was  concluded  in  the 
Project  Office  "that  any  change  to  the  test  program  which  would  give 
the  contractor  cause  to  claim  interference  or  which  would  otherwise 
result  in  renegotiation  of  the  contract  shall  be  avoided". 

A recommendation  based  on  this  conclusion  was  approved  by  the  Secretary 
of  the  Navy  ( SECNAV ) ^ 10 ^ . The  research  accomplished  for  this  study 
has  disclosed  that  no  significant  change  to  the  basic  contract  to 
accommodate  for  early  operational  test  and  evaluation  has  been 
advocated  or  proposed  subsequent  to  SECNAV's  approval  of  the  afore- 
mentioned recommendation. 


The  Cost 

As  was  noted  in  the  first  of  the  quotes  contained  in  the  previous 
discussion  pertaining  to  the  contract,  cost  associated  with  the  addition 
of  early  operational  test  and  evaluation  was  a principle  concern  to 
all  involved.  At  the  time  the  contract  was  awarded,  the  cost  of  the 
operational  evaluation  (OPEVAL),  conducted  by  COMOPTEVFOR  and  scheduled 
for  the  period  1 February  197**  through  1 October  197*+ » was  to  be 
funded  by  CNO  with  the  funds  being  provided  directly  to  COMOPTEVFOR. 

In  addition,  training  for  the  OPEVAL  flight  crews  was  to  be  provided 
by  the  Navy  Test  Pilot  crews  conducting  BIS  trials  during  the  period 
15  October  1973  through  15  June  197*+  from  funds  provided  to  the 
Commander,  Naval  Air  Test  Center  ( COMNAVAIRTESTCEN ) , again  by  CNO. 

In  essence  then,  as  originally  intended,  the  S-3A  project  was  not 
the  source  of  funding  for  operational  test  and  evaluation. 

The  S-3A  project  was,  however,  responsible  for  funding  the 

development  test  and  evaluation  under  which  completion  of  all 

contractual  milestones  was  to  occur  (refer  to  Figure  l).  Although 

project  funds  for  development  test  and  evaluation  were  included  as 

separate  line  items  or  subelements  thereof;  such  as  the  Naval  Air 

Systems  Command  Integrated  Test  Facility  (ITF),  the  original 

cost  breakdown  identified  as  contractor  development  test  and 

(10) 

evaluation  were  by  fiscal  year,  in  millions  of  dollars  : 
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With  the  decision  to  conduct  operational  test  and  evaluation 
earlier  than  originally  scheduled  came  the  requirement  to  train  the 
OPEVAL  flight  crews  earlier  than  origianlly  planned  as  well  as  expand 
the  original  test  and  evaluation  program.  The  Navy  test  crews  who 
were  to  conduct  the  development  test  and  evaluations , the  NPE ' s and 
NPA's,  were  trained  by  the  contractor  at  a cost  to  the  government 
which  was  negotiated  in  the  contract  and  which  is  included  in  the 
cost  figures  above.  These  test  crews,  limited  in  number,  would  have 
neither  the  time  nor  opportunity  to  train  the  OPEVAL  crews.  The 
first  development  configured  aircraft  was  delivered  to  the  Navy  for 
exclusive  testing  in  September  1972,  a date  too  late  to  be  considered 
acceptable  to  start  to  train  OPEVAL  crews.  In  addition,  the 
scheduled  Navy  development  tests  would  have  to  slip  if  Navy  crews 
were  required  to  train  the  OPEVAL  crews. 

Taking  the  foregoing  into  account,  the  only  viable  option  in 
regard  to  training  the  OPEVAL  crews  was  to  provide  contractor  training, 
at  a cost  to  the  government.  It  was  recognized  that  an  expansion  of 
the  originally  scheduled  development  test  program  would  have  to  be 
negotiated  with  the  contractor  in  addition  to  providing  training  for  . 
the  OPEVAL  crews.  When  all  things  had  been  considered,  it  was  decided 
that  the  S-3A  project  would  be  responsible  for  funding  the  early 
initiation  of  operational  test  and  evaluation.  A request  for  quote 
submitted  to  the  contractor  to  accommodate  the  training  of  two  additional 
three  man  flight  crews  and  expansion  of  the  development  test  program 
to  provide  for  approximately  120  hours  of  operational  test  and  eval- 
uation resulted  in  the  contractor's  estimates  of  . *+2  million  dollars 
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for  training  and  1.43  million  for  expansion  of  the  test  program  It 
was  concluded  that  an  estimated  cost  for  early  initiation  of  operational 
test  and  evaluation  of  1.87  million  dollars,  less  than  one  percent  of 
the  total  estimated  cost  of  acquisition  and  ownership  was  acceptable. 

The  Schedule 

The  schedule,  modified  and  expanded  to  accommodate  initial 
operational  test  and  evaluation,  commencing  with  the  first  two  contractual 
milestones,  the  laboratory  avionics  demonstration  and  the  first  flight, 
was  conditionally  approved  by  SECNAV  and  concurred  in  by  DDT&E  in 
early  November  1971.  It  was  then  immediately  submitted  to  the  con- 
tractor for  comment.  In  late  November  1971,  after  about  two  weeks 
for  analysis,  the  contractor  informed  the  government  that  it  was 
willing  to  accept  the  revised  schedule  and  provide  the  training 
required  for  the  OPEVAL  crews  at  it's  previously  quoted  cost.  A 
summary  of  the  revised  schedule  is  presented  in  Figure  2.  Final 
approval  of  the  revised  schedule  by  SECNAV  and  DDT&E  was  accomplished 
on  17  December  1971 ^ It  is  interesting  to  note  that  the  revised 
schedule  did  not  interfere  with  the  contractual  milestones,  an 
objective  in  planning  for  early  initiation  of  operational  test  and 
evaluation,  in  any  way. 

An  Update 

As  of  the  time  this  section  of  this  report  is  being  drafted, 

i 


all  contractual  milestones  have  been  satisfactorily  completed.  The 
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Final  contract  milestone,  delivery  of  the  first  production  configured 
aircraft  to  the  BIS  was  accomplished  on  schedule,  on  9 October  1973. 
All  Navy  development  testing,  the  NPE's  and  NPA's,  and  the  associated 
operational  test  and  evaluation,  have  been  completed  on  or  ahead  of 
schedule.  Using  the  carrier  suitability  demonstration,  which  in- 
volved 288  field  and  shipboard  arrestments  and  39  shipboard  catapult 
launches,  as  an  example,  it  was  completed  three  months  early. 
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COMNAVAIRTESTCEN , who  was  responsible  for  the  development  portion 
of  the  carrier  suitability  demonstration,  reported  (b2)  -that;  during 
the  seven  month  period  of  the  demonstration: 

1.  A total  of  2b  minor  deficiencies  were  discovered,  most  of 
which  were  corrected  by  completion  of  the  demonstration. 

2.  An  absolute  minimum  of  avionics  equipment  and  component 
failures  were  experienced. 

3.  Catapult  flying  qualities  are  "very  good". 

4.  Fuel  and  hydraulic  leaks  were  nonexistent. 

5.  The  S-3A  airplane  "will  be  structurally  and  functionally 
suitable  for  carrier  operations  following  completion  of  BIS 
trials". 

Simultaneously,  from  this  same  carrier  suitability  demonstration, 

COMOPTEVFOR  concluded  That: 

"Based  on  the  'limited'  data,  nothing  was  revealed  during  the 
demonstration  tests  which  would  inhibit  the  operational  suitability  of 
the  S-3A  in  a carrier  environment." 
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The  Quality  and  Consistency  of  Data 

In  contrast  to  the  foregoing  wherein  it  appears  that  both 
COMNAVAIRTESTCEN  and  COMOPTEVFOR  agree  on  the  results  of  the  concurrent 
but  independent  test  and  evaluation,  such  is  not  always  the  case. 

There  have  been  several  instances  of  divergent  conclusions  resulting 
from  the  same  test  or  series  of  tests  and  the  same  data.  An  early 
example  of  this  divergence  came  to  light  as  a result  of  evaluating 
the  flying  qualities  and  performance  in  the  power  approach  configu- 
ration. A Test  Center  pilot  (a  qualified  Navy  Test  Pilot)  acquiring 
data  for  the  NPE,  concluded  that  the  aircraft  characteristics  during 
approach  to  stall  were  satisfactory  and  meet  the  requirements  of  the 
appropriate  specification.  The  QPTEVFOR  pilot,  who  was  not  a 
qualified  test  pilot,  reported  that  the  aircraft  was  unsafe  in  the 
power  approach  configuration  during  approach  to  stall.  The  divergence 
was  resolved  after  it  was  realized  that  the  OPTEVFOR  pilot  had 
accumulated  a total  of  only  eight  flight  hours  of  jet  flying  time, 
all  of  which  was  in  the  S-3A  aircraft.  He  was  comparing  his  reaction 
to  the  S-3A  to  that  of  a propeller  powered  S-2E  in  which  he  had  over 
two  thousand  flight  hours.  His  preliminary  report  was  cancelled 
subsequent  to  his  accumulation  of  more  flight  time  and  consequently  an 
increase  in  his  confidence  in  the  S-3A  aircraft. 

The  most  recent  example  of  divergent  conclusions,  and  for 
which  full  resolution  is  still  pending  at  this  writing,  resulted  from 
a series  of  tests  to  demonstrate  the  compatibility  of  certain  aircraft 
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avionic  components  with  the  Versatile  Avionics  Shop  Test  (VAST) 
system.  Briefly,  the  VAST  system  is  a computerized  system  designed 
to  isolate  faults  in  an  avionic  component.  A total  of  sixty  five 
weapon  replacement  assemblies  (WRA)  were  subjected  to  concurrent 
developmental  and  operational  tests.  With  respect  to  the  development 
test  criteria,  the  VAST  system  correctly  identified  the  fault,  or 
lack  of  fault,  in  thirteen  WRA's,  in  sixty  two  instances.  Although 
the  VAST  failed  to  correctly  diagnose  the  fault  in  three  of  the  WRA's, 
further  analysis  revealed  that  such  incorrect  fault  isolation  was 
within  the  tolerance  of  performance  for  the  VAST  system.  According 
to  the  operational  test  criteria,  however,  seventeen  suitability 
deficiencies  were  recorded,  only  three  of  which  relate  to  the  three 
incorrectly  diagnosed  faults  previously  mentioned.  It  appears  to  the 
Project  Office  that  the  criteria  for  the  two  types  of  test  is 
sufficiently  similar  to  preclude  disagreement  between  results  and 
conclusions.  It  is  the  S-3A  Project  Manager's  conclusion  that  the 
disparity  between  the  results  is  derived  from  the  OPTEVFOR  test 
personnel's  lack  of  formal  training  regarding  test  and  evaluation. 
Additionally,  the  current  OPTEVFOR  test  team  has  only  limited 
operational  antisubmarine  warfare  experience.  He  has  recommended 
that  qualified  Navy  Test  Pilots  and  crews  be  assigned  to  COMOPTEVFOR 
for  satisfactory  completion  of  the  remaining  operational  test  and 
evaluation  of  the  S-3A  weapon  system. 

The  disparity  between  the  results  of  concurrent  development  and 
operational  test  and  evaluation,  is  not  considered  to  be  as  a result 
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of  early  initiation  of  operational  test  and  evaluation.  Divergence 
of  conclusions  between  those  responsible  for  performing  and  reporting 
development  tests  and  those  performing  and  reporting  operational 
tests  have  been  previously  experienced.  But,  in  the  past,  the 
determination  of  operational  unsuitability  has  been  made  after  the 
system  was  in  production;  when  it  was  too  late  to  affect  the  pro- 
duction decision.  On  occasion,  the  determination  of  operational 
unsuitability  has  resulted  in  costly  modifications  to  the  system 
to  insure  that  the  system  provided  the  fleet  with  the  capability 
which  was  to  have  been  provided.  On  other  occasions,  where  modifi- 
cation was  too  costly  or  otherwise  infeasible,  the  fleet  suffered 
reduced  capability,  usually  with  a system  that  was  difficult  or 
costly  to  maintain.  It  was  this  past  history  which  the  Blue  Ribbon 
Defense  Panel  observed  and  to  forestall  similar  occurrences  in  the 
future,  recommended  that  operational  test  and  evaluation  be  initiated 
in  the  acquisition  process  as  soon  as  possible. 

The  divergence  of  conclusions  resulting  from  concurrent  development 
and  operational  testing  does  indicate  a need  for  coordination  between 
the  test  agencies  involved.  A loss  of  independence  is  not  being 
advocated  nor  is  inference  being  made  that  the  conclusions  reached 
by  different  test  agencies  should  be  identical.  However,  if  the 
test  criteria  are  coordinated  and  discussed  by  those  with  similar 
technical  qualifications  to  the  degree  that  the  test  plans  are  coordinated, 
conflict  between  conclusions  may  be  reduced  to  minor  proportions  or 


eliminated  altogether. 


There  is  no  evidence  that  there  was  a lack  of  coordination 


between  representatives  from  COMNAVAIRTESTCEN  and  COMOPTEVFOR . Nor 
is  there  evidence  that  there  was  sufficient  coordination.  There  is 
evidence  that  lack  of  experience,  technical  qualification  and  full 
appreciation  of  the  test  objectives  has  resulted  in  conflict.  More 
deliberate  coordination  may  ameliorate  the  existence  of  conflict  in 
the  future. 
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CONCLUSIONS 
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f It  is  concluded  that  for  the  S-3A  weapon  system  acquisition: 

1.  Incorporation  of  operational  test  and  evaluation  earlier 
than  originally  contractually  scheduled  was  accomplished: 

a.  In  such  a manner  as  to  have  no  effect  on  the  terms  of 
the  basic  contract. 

b.  At  a cost  of  less  than  one  percent  of  the  total 
estimated  cost  of  acquisition  and  ownership. 

c.  By  making  a relatively  minor  revision  to  the  original 

contract  schedule. 

/ 

2.  A possible  lack  of  coordination  between  the  development 

test  agency  and  the  operational  test  agency  has  resulted  in  conflicting 
conclusions  drawed  from  results  of  concurrent  testing. 
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GLOSSARY 


Development  Test  and  Evaluation  (DT&E).  Conducted  to  demonstrate: 
that  the  engineering  design  is  complete;  that  the  design  risks  have 
been  minimized;  that  the  system  will  meet  specifications  and  military 
requirements;  and  that  there  is  assurance  that  it  is  producible 
within  forecast  cost  and  time  limitations.  Performed  by  or  for  the 
developing  agency  which  emphasized  the  technological  and  engineering 
aspects  of  the  system,  subsystem  or  equipment  items.  Normally 
accomplished  under  structured  or  strictly  controlled  conditions. 


„ Operational  Test  and  Evaluation  (OT&E).  Conducted  to  determine, 
for  the  prospective  system:  its  military  utility,  operational 

effectiveness  and  operational  suitability  (including  reliability, 
maintainability  and  logistic  and  training  requirements);  the  need 
for  any  modif ications ; and  requirements , doctrine  and  tactics  for 
its  employment.  It  may  also  provide  data  to  support  or  verify 
material  in  operating  instructions,  publications  and  handbooks. 
Performed  by  operational  personnel  who  focus  on  the  service  suitability 
(including  operational  effectiveness,  reliability,  maintainability 
and  supportability ) of,  and  the  development  of,  optimum  operational 
tactics  for  systems  and  equipment  being  developed  for  service  use. 

Initial  Operational  Test  and  Evaluation  (IOT&E).  That  initial  phase 
of  OT&E  which  is  accomplished  prior  to  the  first  major  production 
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decision  and  must  be  adequate  to  provide  an  estimate  of  expected  system 
operational  effectiveness  and  suitability  (including  reliability, 
maintainability  and  logistic  and  training  requirements).  Pilot  or 
early  production  items  will  be  employed  for  IOT&E  wherever  practiced. 

Concurrent  Test  and  Evaluation ( CT&E ) . Conducted  under  joint 
sponsorship  of  the  developing  agency,  who  is  responsible  for  funding 
and,  in  the  U.S.  Navy,  the  Commander,  Operational  Test  and  Evaluation 
Force  ( COMOPTEVFOR ) . Authority  to  conduct  CT&E  will  be  granted  by 
the  CNO  on  case  basis.  Accomplished  by  combining  the  requirements 
of  developmental,  initial  and  follow  on  operational  test  and  evalua- 
tion providing  the  economies  of  the  combined  approach  outweigh  the 
advantages  of  a more  complete,  independent  DT&E  and  OT&E  program. 

Board  of  Inspection  and  Survey  (BIS)  Trials.  Trials  conducted  by 
or  tinder  the  direction  of  the  BIS  to  determine  contractual  compliance 
with  Navy  specifications  and  conformance  with  mission  requirements 
established  by  the  CNO.  Trials  are  normally  separate  and  distinct 
relative  to  DT&E  and  OT&E. 

Navy  Preliminary  Evaluation  (NPE).  From  one  to  five  phases,  each 
consisting  of  one  or  more  flights,  and  normally  comprise  DT&E. 

Conducted  to  determine:  at  the  earliest  possible  opportunity,  the 

combat  or  mission  potential  and  gross  deficiencies  of  an  aircraft 
and  thereby  enable  an  estimate  of  the  degree  to  which  operational 
requirements  will  be  met;  to  identify  the  need  for  and  allow  for 
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early  correction  of  deficiencies;  to  evaluate  changes  incorporated, 
and;  an  estimate  when  the  aircraft  will  be  suitable  for  BIS  trials. 

Navy  Preliminary  Appraisal  (NPA).  An  element  of  DT&E  developed  for 
the  S-3A  program  to  provide  a cost  effective  method  of  conducting  an 
. assessment  of  the  antisubmarine  warfare  (ASW)  avionics  suite  using 

a P-3C  aircraft  as  a flying  test  bed.  Specifically,  conducted  to 
determine:  a confidence  level  that  final  design  objectives  for  the 

S-3A  weapon  system  can  be  met  for  the  ASW  mission;  to  obtain,  in 
an  operational  environment , a preliminary  operational  evaluation  of 
the  avionics  suite,  and,  to  perform  a technical  assessment  of  the 
equipment  prior  to  installation  in  the  S-3A  aircraft. 


